One Needs to Be Careful When Dismissing Outliers: A Realistic Example

Traditional approach to eliminating outliers is that we compute the sample mean μ and the sample standard deviation σ, and then, for an appropriate value k0 = 2, 3, 6, etc., we eliminate all data points outside the interval [μ−k0·σ, μ+k0·σ] as outliers. Then, we repeat this procedure with the remaining data, eliminate new outliers, etc., until on some iteration, no new outliers are eliminated. In many applications, this procedure works well. However, in this paper, we provide a realistic example in which this procedure, instead of eliminating all outliers and leaving adequate data points intact, eliminates all the data points. This example shows that one needs to be careful when applying the standard outlier-eliminating procedure. 1 Formulation of the Problem Need to eliminate outliers. In the traditional approach to data analysis, based on the sample, we estimate the means of the corresponding quantities, we estimate the variances, covariance, and correlations; see, e.g., [3]. This usually works well, but sometimes, we have outliers, i.e., values caused, e.g., by the malfunctioning of the measuring instrument. Outliers ruin the estimations. For example, if we are interested in the average temperature, and in addition to 100 measurement results around 20◦ C, we have a (clearly erroneous) value 1000◦ C, then the sample average x becomes x ≈ 20 + . . .+ 20 (100 times) + 1000 101 ≈ 30.